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ABSTRACT 

The decision process required for Individually 
Prescribed Instruction (IPI) , an adaptive instructional program 
developed at the University of Pittsburgh^ is described. In IPI, 
short tests are used to determine the level of proficiency of each 
student in precisely defined learning objectives. The output of these 
tests is used to guide instructional planning for individual 
students. The nature and effect of errors in proficiency decisions 
are described and a procedure for reducing the probability of such 
errors is proposed. The plan calls for a Bayesian procedure which 
would incorporate prior information on the instructional program^ for 
example the distribution of the percentage of items answered 
correctly by students. Such a procedure would permit inferences about 
the true level of functioning of each student. The final section of 
the paper proposes two methods for implementing these procedures in 
an ongoing IPI program: one approach calls for the integration of the 
procedure as a part of a computer-based instructional management 
system^ whereas the second approach describes how the procedure can 
be made tractable in a typical, non-automated IPI classroom. 
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ABSTRACT 

The decision process required for individually Prescribed Instruction (IPI), an adaptive 
instructional program developed at the University of Pittsburgh, is described. In IPI, short tests 
are used to determine the level of proficiency of each student in precisely defined learning 
objectives. The output of these tests is used to guide instructional planning for individual 
students. 

The nature and effect of errors in proficiency decisions are described and a procedure for 
reducing the probability of such errors is proposed. The plan calls for a Bayesian procedure 
which would incorporate prior information on the instructional program, for example the 
distribution of the percentage of items answered correctly by students. Such a procedure would 
permit inferences about the true level of functioning of each student. 

The final section of the paper proposes two methods for implementing these procedures in an 
ongoing IPI program: ono approach calls for the integration of the procedure as a part of a 
computer-based instructional management system, whereas the second approach describes 
how the procedure can be made tractable in a typical, non-automated IPI classroom. 
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IMPLEMENTATION OF A BAYESIAN SYSTEM FOR DECISION ANALYSIS 
IN A PROGRAM OF INDIVIDUALLY PRESCRIBED INSTRUCTION 



Richard L, Ferguson 
Melvin R. Novick 



INTRODUCTION 



The feasibility of instructional programs designed 
to adapt to the individual n^eds of learners has been 
adequately demonstrated by educational systems 
like Individually Prescribed Instruction (Glaser, 
1968) and A Program for Learning in Accordance 
with Needs (Flanagan, 1967). Although these 
programs accomplish individualization in somewhat 
different ways, each includes components which 
can be described by the following sequence of 
operations: 

1. Specification of the learning objectives in terms 
of observable student behavior. 

2. Assessment of the student's entering compe- 
tencies. 

3. Assignment or election of educational materials 
and/or experiences fitted to the student's 
individual needs. 

4. Continucjs assessment and monitoring of the 
student's performance and progress. 

Since programs like 1 PI and PLAN call for adaptation 
of the learning environment to meet individual 
requirements, they necessarily rely heavily on the 
systematic assessment of student progress. Glaser 
(1 968) has observed that, in I PI, test data serve as the 
primary source of information enabling teachers to 
^ make differential decisions regarding student 
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instruction. Thus, steps (2) and (4) play a prominent 
role in the successful Implementation of IPI. A 
review of current decision-making procedures for 
four selected individualized instructional programs 
has been given by Hambleton (1973). 

The fundamental purpose for testing in in- 
dividualized instructional programs like IPI and 
PLAN is to ascertain whether or not the student has 
attained some prescribed level of proficiency in a 
specified learning objective. Hambleton and Novick 
(1973) have observed that, "Questions of precise 
achievement levels and comparisons among indi- 
viduals on these levels seem to be largely irrel- 
evant." Because test data are used initially to place a 
student at the appropriate point within an 
instructional program or sequence, and thus to 
identify appropriate learning materials or ex- 
periences given his needs, the test models which 
have emerged to serve this function are very dif- 
ferent from those used for standard insi actional 
models. Because these tests relate a studr it's per- 
formance on items drawn from a carefully ; oecified 
domain to a prespecified criterion orstandard, these 
tests have come to be called domain or criterion- 
referenced tests. 

It is not the purpose of this paper to contrast the 
differences between norm-referenced tests and 
criterion-referenced tests. Suffice it to say that 
criterion-referenced tests are deliberately con- 



structed so as to yield measurements which are di- 
rectly interpretable In terms of specified per- 
formance standards (Glaser and Nitko. 1971, The 
process of constructing such tests involves the 
specification of a domain of tasks that the student 
should be able to perform and the selection of 
samples of these tasks representative of that do- 
main. The students competency in the skill is 
judged in terms of his performance in responding to 
the sample ofthe tasks which is drawn. Performance 
on this sample is used to infer that his level of 
functioning in the domain either does or does not 
meet sofne prescribed standard. 

Because student performance on tests used in IPl 
and PLAN is used as the basis for making decisions 
affecting placement and advancement, and because 
it is crucial that these decisions be accurate, major 
importance is attached to the precision with which 
each person's true domain score (level of 
functioning) can be related to the prescribed 
proficiency level. However, due to ttme constraints, 
the tests are often comprised of a very small number 
of items, usually 1U or less. Thus, the precision of 
judgment from such tests must be open to question. 
Because of the important role which testing plays in 
the instructional decision making within IPl, im- 
provement in the quality of the decision process 
would oe greeted with considerable enthusiasm if \X 
could be accomplished without a corresponding in- 



crease m the length of the tests. This paper is 
addressed to the problem of showing preciseh "oow 
some new developments in statistical theory make 
this goal attainable. More specifically, the present 
paper indicates precisely hovj these Bayesian 
methods could be Integrated Into an ongoing IPl 
program. In order to lay a proper foundation, one 
describing the exact nature of the measurement 
problem in IP!, we propose to confine discussion to 
one major component of the system, the math- 
ematics program. To this end, a general de- 
scription of the assessment instruments used in IP! 
mathematics is contained in the next section. 

The mathematical and statistical models which 
form the basis of the proposed application, and the 
outline of this application, are based on the work of 
Novick, Lewis, and Jackson (1973), and the ampli- 
fications contained in Lewis, Wang, and Novick 
(1973), Wang (1973), and Wang and Lewis (1973a, 
1973b). A theoretical discussion of these methods is 
contained in Hambleton and Novick (1973). The 
Bayesian methods of statistical inference developed 
in these papers combine direct observation 
information on each student with certain 
background information, to permit more accurate 
decision-making than would be possible withoutthe 
use of this background information. The use of this 
background information makes possible the gain in 
accuracy without additional testing. 



THE IPl MATHEMATICS PROGRAM 



Ferguson (1970a) provides a detailed description 
of the IPl Mathematics program. Highlights of that 
description are provided in subsequent parts of this 
section. I.n particular, attention is given both to the 
structure^f the curriculum and to the test model 
which plays such an important role in the 
management of the program. 

The Curriculum 

Figure 1 conveys the general organization of the 
mathematics curriculum. Ten content areas, 
Numeration/Place Value, Addition/Subtraction, 
Multiplication, Division, etc., are identified; each 
occurring at various levels of difficulty. The ten 
areas are listed in a hierarchical order that is 
followed in instruction. The intersection of each 
level with a specific content area determines a unit 
that consists of a set of behaviorally defined 
objectives or skills. Each number in the table 
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Indicates the number of skills in the unit. Thus, E 
level-Systems of Measurement is a unitthat consists 
of a set of five behavioral objectives (skills) which 
ihare a similar content but are less difficult than the 
skills contained in the F. level-Systems of Mea- 
surement. The absence of a number at any position 
in the chart indicates that no unit exists for the 
corresponding content area and level. Atthe bottom 
of Figure 1, we have listed the specific behavioral 
objectives for E level-Systems of Measurement. 

The Test Model 

As previously indicated, the assessment instru- 
ments in IPl perform a dual role in the program, 
serving both a placement and a diagnostic function. 
The tests are placement oriented in the sense that 
they locate a student's position in the curriculum 
with respect to theskillsfor which he lacks sufficient 
proficiency, but for which he has the necessary 



Level 





A 


B 


C 


D 


E 


F 


G 


Numeration/Place Value 


15 


9 


14 


5 


6 


7 


6 


Addition/Subtraction 


17 


12 


13 


10 


4 


4 


6 


Multiplication 




4 


7 


9 


7 


4 


3 


Division 




3 


4 


7 


9 


5 


6 


Fractions 


3 


3 


6 


■/ 


11 


8 


8 


Money 


. 1 


1 


5 


5 








Time 




6 


6 


4 


4 


2 




Systems of Measurement 




3 


6 


6 


5 


5 


6 


Geometry 




3 


2 


4 


6 


4 


2 


Applications 




3 


8 


9 


5 


4 


6 



Behavioral Objectives 

E LevehSystems of Measurement 

1 . Given a ruler, the student measures a line segment with the indicated degree of precision. LIMIT: smallest unit 
of precision 1/8 inch; line segments to 10 inches. 

2. Given 20 cut-out regions that are each 1-inch squares and an illustration of a rectangular region, the student 
uses the 1-inch squares to determine the area of the given rectangular region. LIMIT: areas < 20 square 
inches. Length of sides of rectangles must be multiples of 1 inch. 

3. Given the measures of the sides of a rectangular region, the student determinestheareaofthatregion. LIMIT: 
integral measures; one unit of measure per problem; un'ts of measure— inches, feet, yards, miles. 

4. Given the measure of the sides of a rectangular region, the student determines the perimeter and the area of 
that region. LIMIT: At least one of the measures (length, width) must be integral; both measures must 
be < 100; one measure may be a common fraction < 1 with denominator < 10; 1 unit of measure per problem; 
units of measure— inches, feet, yards, miles. 

5. Given a weight measurement, the student coriipletes a statement to show an equivalent measurement in a 
different unit of weight measure. Given a word problem that requires conversion of a given weight 
measurement expressed in standard units to an equivalent weight expressed in another standard unit, the 
student solves the problem and writes the answer with the appropriate label. LIMIT: units— ounces, pounds, 
tons. 



Fig. 1. Matrix of Units in the IPI Mathematics Curriculum. 



prerequisite skills so that he can begin work. The 
same tests are diagnostic in that they provide 
information that identifies sk'lls in which thestudent 
has not achieved sufficient proficiency and also 
provide insight as to specific facets of theseskills on 
which instruction is required. A review of the various 
tests utilized in the matherriatics program follows. 

Curriculum Placement Tests 

Upon entrance to the mathematics program, the 
placement tests provide a global picture of each 
student regarding his level of proficiency with 
respect to the skills in each unit of the curriculum. 
The data generated by the placement tests are used 
to develop a profile for each student indicating those 
units in which he has'sufficient proficiency in all of 
the skills and those in which he has insufficient 
proficiency. For example, the outcome of a 
placement test might yield a profile indicating 
sufficient proficiency in all of the skills in level D of 
the curriculum, and insufficient proficiency in the 
skills of units at a higher level of difficulty. In this 
case, the student would begin work in units at level E 
of the curriculum. More typically, a student might 
demonstrate proficiency at level D-Numer- 
ation/Place Value, level F-Addition/Subtraction, 
level E-Money, level C-Time, and perhaps *evel D in 
all other areas. Such a student would probably then 
begin instruction in level C-Time, this being the 
lowest level in the area hierarchy at which 
instruction is prescribed. 

Because of the global nature of placement tests, 
they must assess a very large domain of math- 
ematics skills. Consequently, practicality demands 
that the tests include only a small number of items 
on key objectives in each unit of the curriculum. 
Thus, important placement decisions are 
necessarily dependent on tests with a small number 
of items. 

Unit Pretests 

Once a piacementtest has been used to determine 
e profile for a student, a decision can be made, as 
indicated in the previous section, regarding the unit 
on which the student begins his work. At this point, a 
unit pretest is administered to identify the specific 
objectives in. the unit for. which the student has 
sufficient (insufficient) proficiency. Each pretest 
consists of several short subtests, one for each 
objective in the unit. 

It is possible for a student to demonstrate 
sufficient competency on all objectives in the unit. If 
this were to occur, the student would continue 



working at the same level, but proceed to the next 
unit in the area hierarchy where he would be given 
another unit pretest. Thus, the pretest provides 
additional information about a student, information 
which is focused at the level determined by the 
placement test. 

The pretest decision can and sometimes does 
override a part of the placement decision. This 
occurs when proficiency is demonstrated by the 
student in areas and at levels not indicated by the 
placement test. Thus> the IPl testing paradigm 
initially involves a two stage semisequential testing 
program with the placement test largely determining 
the level at which more intensive testing is to take 
place. 

After the unit pretest has identified the specific 
skills for which the student requires instruction, 
student test performance on each of these 
objectives is examined by the teacher to identify 
particular types of errors or patterns of errors. In this 
manner, learning materials and/or experiences 
consonant with the individual's needs can be 
prescribed. 

The typical pretest includes between six and 
(preferably) ten items for each objective. Obviously, 
the size of the domain of items varies with the 
particular skill. Usually, however, the domain is 
quite large. Thus, important instructional decisior:s 
are often based on student performance on a small 
number of items that have been representatively 
sampled from a very large domain. The relative 
shortness of the tests can certainly be justified from 
a practical point of view. Longer tests might be 
considered repressive and would certainly exceed 
reasonable bounds in terms of the proportion of time 
given over to them within the total instructional 
process. Thus, it would appear that the key to more 
effective and more reliable decisions lies not in 
increasing the length of the tests beyond, say. eight 
or ten items, but rather in making better use of the 
data available within the present system. 

Curriculum Embedded Tests 

These short ''quizzes" measure the student's level 
of proficiency in a single skill within the curriculum. 
The written instructional material for each skill in a 
mathematics unit contains two curriculum em- 
bedded tests (CETs). The tests are self-evaluation 
devices used by the student as a check on his 
progress as it relates to his work on a given skill. 
Thus, the student who has completed several 
learning activities related to the development of his 
proficiency in a particular skill might take a GET to 
determine whether he has attained sufficient 



proficiency at this point or whether he needs to 
complete additional steps /n the instructional 
process. 

The GET typically consists of from four to six 
items. Because these short tests serve primarily as 
self-checks for the student, and because no crucial 
instructional decision is dependent upon student 
performance on these tests, they seem to be ade- 
quate for the task which they serve. 

Unit Post tests 

These instruments are equivalent forms of the unit 
pretests. They are generally administered after the 
student has concluded learning activities for all 
skills for which he was identified as being 



insufficiently proficient on the unit pre 'ist. On the 
basis of the student's performatice on the posllesl. 
he is either advanced to the next unit or rejuired to 
work withadditional instructional materials on those 
skills for which his test performance did not indicate 
that he achieved a sufficient level of proficiency. A 
student generally does not advance to a new unit 
until he has demonstrated sufficient proficiency for 
all objectives of the current unit. 

As with the pretests, decisions resulting from an 
analvsis of posttest data rely upon tests which gen- 
erally contain a small number of items. Because in- 
correct proficiency decisions can be detrimental to 
the student's progress, a procedure which could add 
substantially to the accuracy of the decision with- 
out increasing the length of the test would be most 
worthwhile. 



THE INSTRUCTIONAL 

In this section, the process by which test data are 
used to make instructional decisions is briefly 
summarized. In addition, a discussion of the nature 
and consequences cf decision errors resulting from 
the analysis of test data is presented. 

A Summary of the Decision Process 

Gross placement tests which sample a broad 
cross section of the important skills in each unit of 
the mathematics curriculum are administered upon 
each student's entry into the I PI program. Score data 
resulting from these tests are used to determine a 
profile suggesting the student's level of proficiency 
in each content area of the curriculum. 

At this point, the student completes a pretest for 
the first unit in the curriculum continuum in which 
his level of proficiency is insufficient. The profile 
resulting from the pretest identifies those skills for 
which learning materials and/or experiences are • 
required if the student is to achieve the specified 
level of performance. During the instructional 
process, curriculum embedded tests are available to. 
the student as a means of self-evaluation and an 
estimate of progress as he works on the skills. After 
he has completed work on a'l skills in the unit and is 
satisfied that he has sufficient competency in all of 
the unit skills, he is administered a posttest which 
verifies his progress or identifies those skills for 
which additional instruction is indicated. Once the 
unit is successfully completed, the student ad- 
vances to the next unit on his prescription where he 
is administered a pretest and the cycle is repeated. 



DECISION PROCESS 

The Nature and Effect of Decision Errors 

The placement tests, pretests, and posttests are 
used primarily to verify that a student either has 
sufficient proficiency. I.e. mastery, in a given set of 
skills or that he has an inadequate level of pro- 
ficiency inthoseskills.Clearly.it is desirable that the 
mastery decisions for a student be as accurate as 
possible. The importance of accuracy of the mastery 
decision for a student is perhaps best emphasized 
by a discussion of the consequences of an incorrect 
decision. 

As previously indicated, the IPl tests are con- 
structed by sampling items from the domain of items 
for the objectives included on the tests. Since any 
sampling which does not exhaust the population of 
items for an objective can lead to an incorrect 
mastery decision and since exhaustive testing is 
impossible, it is necessary to tolerate the risk of 
making wrong decisions. In an IPl context, a Type I 
( a ) error occurs when an examinee has sufficient 
proficiency in a skill but the outcome of the testing 
suggests that he does not. As a result, he is 
prescribed work lessons which may serve no sig- 
nificant function. A Type II ( i3 ) error occurs when- 
ever the examinee, in fact, lacks proficiency in askill 
but on the basis of test results is said to have 
sufficient proficiency. The consequence of a Type 1 1 
error is that needed remedial instruction is not 
provided. A Type II error is perceived to be poten- 
tially more serious than aTypel errorsincetheType 
II error could easily result in the student having 
difficulty proceeding through a unit and might 



eventually lead to an impasse in instruction: 
whereas, the Type I error wil! at worst require that 
the student pursue a review-like study of skills in 
which he is already proficient. 

Although it is clear that the magnitude of the 
consequences of an incorrect proficiency decision 
for a student varies with the direction of the error, it 
is eqtyally clear that in both cases the error may have 
detrimental effects for the student. The fact that the 
tests on which these decisions are based have a 
small number of items per skill suggests that such 
errors probably occur quite frequently. Given the 
constraints imposed by a program which already 
has a heavy testing component, increasing the 
length of the tests is not a tractable method for 
achieving increased accuracy in the mastery 
decision process„ However, it may very well be 
possible to incorporate additional information into 
the decision process and thus improve the overall 
accuracy of the decisions being made. It is this 
hypothesis to which the remainder of this paper is 
addressed. 

In IPI, as in all individualized instructional 
programs, decisions are focused around the 
individual student. If a statistical procedure that 
uses information other than that contained in the 
immediate direct observations on the studeni is 
contemplated, then a Bayesian procedure incor- 
porating prior information on each student comes to 
mind. This information would consist of results of 
the student's performance on previous instructional 
units. In this way, interindividual variability on prior 
test performance would be helpful in making current 
decisions. 

The problem with this thinking is that the entire 
thrust of individualized instruction works toward a 
reduction of interstudent variability of test results. A 
student moves ahead to a new unit of instruction 
only when, it is thought, he is prepared to do so. 
Indeed, he is encouraged not to take the unit 
posttest until there is strong evidence that he is 
prepared to perform well on it. A great deal of 
posttest score variability is in fact observed, but 
much of it, though not all, results from unreliability 
due to the necessarily short length of these tests. 
Thus, realistically, there is little or no useful 
differential prior information about the individual 
student. 

On the other hand, there is a great deal of 
information available about the instructional 
program. Quite specific information is available 
concerning the distribution of the percentage of 
items answered correctly by students (Novick, 
Lewis, and Jackson, 1973), and it is thus possible to 
make inferences about the true level of functioning 
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of each student, and the mean and standard 
deviation of these true values in the population of 
students. Of course, if the instructional programs 
were completely efficient and the students were 
without human frailties, there would be no variation 
in true levels of functioning of students on posttests. 
A student would remain in a unit on^y until that 
instant at which his level of functioning attained the 
prespecified criterion. Nothing approaching this is 
possible with present instructional technology. 
However, if we knew this were the true state of 
affairs, then we would ignore individual test scores 
and use our information on the group mean and 
variance to mr ';e a positive proficiency decision for 
all students. 

In the real world of Individually Prescribed 
Instruction there will be some variation in true levels 
of functioning among students on posttests. The 
delicate manner in which background information is 
combined with the direct observational data in the 
Bayesian decision process, and the increment in 
decision-makinp accuracy resulting therefrom is 
detailed in Novick, Lewis, and Jackson (1973) and 
Lewis, Wang, and Novick (1973). 

Finally, we may note one additional source of 
background information that can be utilized when, 
as in IPI, testing involves joint measurement on 
several skills, simultaneously. In this situation and 
assuming some relationship among the skills, it is 
possible to use the collateral information contained 
in the t - 1 of t tests scores for each person to help 
estimate each t-th test score. Thus, if a person 
scored highly in t - 1 subtests and a little less highly 
in thet-th, we would suspect that this might be due /n 
part to bad luck or carelessness, and we would be 
inclined to make some adjustment inourestimateGf 
his proficiency on that skill. The Bayesian theory 
and methods described by Wang and Lewis (1973a, 
1973b) provide the rationale and prescription for 
doing this. 



Implementation Procedures 

The decision analysis procedures employed by 
teachers and students in the IPl program must not 
be overly complex. Thus, the final output ot the data 
analysis procedures used to judge the level of 
proficiency of a student must be so simple that 
teachers, aides, and even students can read the 
results, interpret them, and then take whatever 
action is indicated. It will be permissible to use 
sophisticated statistical methods, but teachers, 
aides, and students must not be required to 
understand much more than is contained in this 



paper. In shor*. although it is not necessary that 
'eachers and students understand the details of the 
analysis, they must be provided information which 
facilitates (heir instructional decision making. In the 
following section, procedures for dealing with the 
preceding concerns are discussed. 

"^^76 coHection and anaiysis of data. During the 
past several years, considerable investigation has 
been underway into the feasibility o using a 
computer as an integral part of the IPI program. A 
thorough discussion of the most recent devel- 
opments is available in a progress report (Block. 
Carison, Fitzhugh, et al., 1973) recenWy released by 
the Learning Research and Developmer»t Center at 
theJJni versify of Pittsburgh. Earlier reports include 
Cco-^ey and Glaser (1969), Ferguson (1970b, 1971), 
and Ferguson and Hsu (1971). The activities 
described in these reports emphasize somewhat 
visionary ideas for how the computer can best be 
employed in an individualized program of 
instruction. Although these studies i nclude the more 
conventional modes of computer-assis;.. 
instruction, they extend far beyond into such areas 
as computer testing and instructional management. 

It is in this latter area, instructional management, 
that Bayesian procedures for determining pro- 
ficiency decisions would best seem to reside. Work 
in this area has been concerned with how the 
computer can assist in the planning and subsequent 
monitoring of both short- and long-term instruction 
for individual students. Thus, it would seem 
appropriate to incorporate a decision-making pro- 
cedure concerned with individual proficiency level 
in some skill, or set of skills, as an element of the 
instructional management component of the IPi 
program. Specifically, the computer might be used 
to receive test data on a student and combine this 
with previously acquired information on other 
students in this IPI program, analyze the data using 
Bayesian analysis techniques, and then print out a 
report indicating the confidence which one could 
place in deciding that the student is proficient in a 
given skill at some prespecified level of 
performance. A more detailed discussion of how this 
procedure might work is now provided in the context 
of IPI posttests. Procedures similar to those 
described^ below would apply for placement tests 
and pretests as well. 

Development, and use of a pastiest profile. The 
primary purpose for administering a placementtest, 
a pretest, or a posttest is to acquire data which can 
be used to evaluate a student's Instructional needs. 
When a student is administered a posttest, he is 
presumed to have had instruction ?n those skills for 
which he lacked sufficient proficiency at the time he 



was administered the unit pretest. The posttest 
eitner affirms the student s success in acquinng the 
skills or calls attention to those skills m which 
additional work is required before he can proceed to 
the next unit. Thus, the only information which the 
teacher and student need is a simple statement 
regarding the level of proficiency at which the 
student has perfonned on each skill in the unit. 
Figure 2 shows an IPI posttest profile based on atest 
consisting of five, eight-item subtests, each 
measuring proficiency level on a particular skill. 



Level E-Mu!tiplication/Division 



Skill Percent Correct 



1 87.5 

2 87.5 

3 75.0 

4 100.0 

5 67.5 



Fig. 2. Sample of Posttest Profile Currently in Use 

in (PI. 



Presently, the posttest profile names each skill in 
the unit and lists the percentage of items which the 
student answered correctly. Given the sample 
profile in Figure 2 and a criterion (cutoff) score of 
85%, it is likely that the student would be called upon 
to undertake additional work in the 3rd and 5th skills 
of the unit. 

Under the proposed change, rather than 
evaluating student proficiency solely on the posttest 
results, additional data would be incorporated 
within the decision analysis process, and 
furthermore, the quantity reported would be an 
index relating thestudent's estimated proficiency to 
a stipulated standard. However, it should be 
emphasized that although the nature of the data 
reported in the student profile v/ould change, the 
procedures employed by the teacher and/orstudent 
to judge proficiency would remain the same. 
Specifically, the posttest profile, which presently 
contains a statement of the percentage of items 
correctly answered fcr each skill, would be altered to 
report the probability that the student has achieved 
some prespecified level of proficiency in each 
objectivi^ As far as the teacher or student is 
concerned, the proficiency decision process is 
exactly the same— judgments are based on the 
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evaluation of a sing'e number or "index" for each 
skill. Figures provides an example of such a profile. 



Level E-Systems of Measurement 

Skill Mastery Index 

1 80 

2 90 

3 76 

4 92 

5 40 



Fig. 3. Proposed Sample Posttest Profile Using 
Bayesian Decision Analysis Procedures. 



In Figure 3, the column labeled Mastery Index 
actually represents a probability statement. If, for 
example, the criterion or cutoff score for sufficient 
proficiency is .85, the Mastery Index column gives 
the probability that the student's level of proficiency 
is above .85 for each skill. In this case, the mastery 
index for skill 1 is .80. We see that the actual test 
performance was only 75%. This m ight suggest, very 
roughly, a probability of .50, a 50/50 chance, for the 
true level of functioning being above .75. However, 
the Bayesian analysis, using the collateral 
information has raised to .80 the probability that the 
student's level of functioning is above .85. 
Therefore, if we would want to move a student on if 
the odds were better than three to one in favor of his 
actually being proficient, we would advance this 
student since his probability of mastery is greater 
than .67. 

Implementation mode. A profile similar to the one 
described in Figure 3 could be provided in at least 
two ways. One method of delivery would requirethe 
availability of tests which are administered by 
computer. Presently, test administration by 
compute" is very much a part of the feasibility study 
underway in IPI. Given the existence of a unit 
posttest on some specified unit, it would seem quite 
possible for sample data generated bythe computer 
test to be merged with a file containing collateral 
data on student success in the systenr.. For example, 
the computer test program could be designed, upon 
student completion of the test, to call a subroutine 
which would access the collateral datafile, combine 
the two sets of information, compute the mastery 
indices (aposteriori probabilities), and print out a 
profile similarto Figure3. In this case, the collateral 



data would be in a file permanently maintained on 
the computer and periodically updated. This 
function could be performed automatically by the 
computer. 

Since it is very likely that many schools using IPI 
will not have ready on-line access to a computer, an 
alternative procedure for providing the same 
decision analysis would call for the construction of 
simple "Mastery Index" tables. These tables would 
permit the teacher, the aide, or a student to 
determine the probability that the student has 
sufficient proficiency in a skill by simply entering the 
table with the number of items answered correctly 
on each skill of the posttest. Figure 4 serves as an 
example of such a table. 



Level E-Systems of Measurement 

Skill 1 

Number of items 
answered correctly Mastery Index 



8 


.98 


7 


.93 


6 


.85 


5 


.73 


4 


.60 


3 


.34 


2 


.27 


1 


.12 


0 


.03 



Fig.4.SampIeof Proposed "Mastery Index" Table 
for IPI. 

Given knowledge of the number of i'ems which 
the student answered correctly out of a possible 
eight on skill 1 of the level E posttest for Systems of 
Measurement, the teacher or student would enter 
the "Mastery Index" table with that number. For ex- 
ample, if the student responded correctly to seven of 
eight items, he would enter the table in the left hand 
column with the number seven and consequently 
determine that the probability that the student has 
the prespecified level of proficiency, say .85, is 93. 
The decision as to whether to move a student 
forward or not would depend on this probabilityand 
the relative disutilities associated with the two kinds 
of errors. The simple methods for accomplishing 
this are described by Davis, Hickman, and Novick 
(1973). 



The indices reported in the tables would have 
been generated at some earlier time and would have 
included consideration of relevant prior data 
regarding student success on the skills contained in 
the unit. The tables would be updated on a regular 



basis as increased numbers oi students procG?eded 
through the system, thus making more prior 
information available. Such an updating might 
occur once or twice a year. 



SUMMARY 



Individualized learning programs like IPI generate 
substantial amounts of data related to student 
success on skills in the system. Given these data, it 
seems reasonable to suggest that they should be 
used to improve the quality of instructional decision 
making. In particular, priordata should be combined 
with sample test data to form a more complete 
information base on which to evaluate student 
proficiency. By using such data jointly, instructional 
decisions regarding a student's needs as they relate 
to a given skill or set of skills will be deserving of 
more confidence than present decisions which are 
currently based solely on the student's performance 
on a short test. 



Two procedures for implementing such a plan 
have been proposed. One calls for the marriage of 
the Bayesian decision analysis procedures with 
computer administered tests; whereas, the other 
would rely on the teacher or student to consult a 
table to translate student test performance to a 
"Proficiency Index" which would incorporate both 
the test data and prior data regarding student 
si:ccess in the system. The ultimate criterion for 
success of such a plan is the extent to which it leads 
to improvements in the instructional decision 
process. To this end, the next step is to implement 
the procedures and evaluate their impact on 
students within IPl. 
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